Motion pictures are composed of a sequence of image frames displayed to viewers at a fast frame rate. The perceived image resolution is a key indicator for the exhibition quality of a motion picture, and it is a combined result of both spatial resolution and temporal resolution. The spatial resolution measures the level of details within each image frame that can be perceived by audience, and it is determined by the quality of the display system as well as by the quality of motion picture content. The temporal resolution of a motion picture measures the level of motion smoothness of a moving image sequence, and it is determined by the frame rate at which the motion picture images are displayed. For cinematic presentations, the standard frame rate of a conventional motion picture is 24 fps (frames per second). However, there exists a number of higher frame rate motion picture formats. An example of presenting a motion picture digitally at a higher frame rate is generally described in U.S. patent application no. 2002/0149696 as a method of temporally interpolating motion images to a higher frame rate so the motion picture can be presented digitally at the original frame rate or a higher frame rate. The frame rate interpolation method may rely on motion vector analysis, such as the analysis used in Kodak's Cineon System. However, such system does not provide for digitally re-mastering an entire motion picture or producing acceptable image quality for cinematic or large format cinematic applications that demand relatively high visual and audio quality. For example, the system does not provide for artifact repair or sufficient processing speed to meet day and date release schedules.
Other examples of such high frame rate motion picture formats include Showscan (60 fps), ToDD-AO (30 fps) and IMAX® HD (48 fps). IMAX® HD is a 15-Perf/70 mm film format that captures and displays a motion picture at a frame rate of 48 fps. The first IMAX® HD film Momentum was produced by the National Film Board (NFB) of Canada and premiered at EXPO92 in Seville, Spain, in 1992. The study on the IMAX® HD technology was subsequently presented to 135th SMPTE Technical Conference in 1993. The study indicated that, compared with a standard IMAX® format using a frame rate of 24 fps, IMAX® HD dramatically improves image realism by enhancing clarity and sharpness, reducing film grain noise and virtually eliminating of motion artifacts like strobing and motion blur. The study further indicated that, even for still shots, the perceived image resolution was “notably greater than standard IMAX® (format)”. The study provided evidence that temporal resolution enhancement through frame rate increase could improve perceived image resolution. Similar resolution improvement effects were later reported from the experiment work on two other 48 fps film-based projection formats. One of such format was a 3-Perf/35 mm format called MaxiVision, and the other was a 5-Perf/70 mm format called Super Dimension-70 (SDS-70) by Super Vista.
Over the past decade, relatively few motion pictures have been produced and exhibited at a frame rate higher than 24 fps. There are both economical and technical limitations that prevent a film-based motion picture from being produced at a higher frame rate. On the production side, shooting at a higher frame rate increases the film costs and production costs. More lighting may be needed on a set due to reduced exposure time as the result of using a higher frame rate, which contributes to the production cost increase. On the exhibition side, projecting at a higher frame rate significantly increases the complexity and the cost of a film projector as well as the cost of film prints. Because of those limitations, neither IMAX® HD nor other proposed higher frame rate film formats became financially viable for mainstream motion picture productions.
The advance of digital projection technology makes it possible to economically exhibit motion pictures at a higher frame rate. The Digital Cinema System Specification recently released by Digital Cinema Initiatives (DCI) includes 48 fps as a projection option. However, the cost of producing a motion picture at a 48 fps remains relatively high. One solution is to enhance the temporal resolution of a motion picture by converting the images to a higher frame rate. A frame rate conversion method actually creates synthesized image frames digitally based on the original image frames. Over the past decades, a number of frame rate conversion methods were developed for motion pictures and for video format conversion. These methods range from simple frame (field) repeating, frame (field) averaging to more complex methods such as motion-compensated frame interpolation. A motion-compensated (MC) method analyzes the motion of image elements across neighboring image frames and creates a synthesized new frames based on the estimated motion information. An MC method usually produces smoother motion than other methods.
A typical MC method has a motion estimator that calculates the movement of each image element of an image frame with respect to adjacent frames. An image element can be defined as a single pixel, a block of n×m pixels or a group of pixels describing an object. A single motion vector is normally used to indicate the direction and the strength of the movement of an image element from a present frame to a future frame. Sometimes, a pair of motion vectors is used to indicate the movement of an image element both from a present frame to a future frame and from the future frame back to the present frame. This is called bi-directional motion estimation. Motion vectors may not be sufficient to describe the movement of a group of pixels describing an object because the shape of the object may also change from a present frame to a future frame. In such a case, some forms of mathematical description of object shape warping may also be included along with motion vectors. There has been a plethora of MC methods proposed over the last decade for video format conversion. A majority of those methods can be fully automated with little or no need for human intervention. However, none of those methods are capable of producing adequate image quality required for motion picture applications.
Some algorithms have been proposed for converting a motion picture to a video format at a field rate of 50/60 fields per second. Such applications typically require fully automated algorithms ranging from standard 3:2 pulldown to motion-compensation based frame rate conversation (MCFRC) algorithms. An MCFRC algorithm may create better image quality and smoother motion but it also produce other artifacts that result from motion estimation errors. The MCFRC algorithm generally includes three categories: (1) block-based methods; (2) object-based methods; and (3) pixel-based methods. The block-based methods can be implemented using common block-based motion estimation algorithms similar to those in MPEG and H.264/AVC codecs. The object based methods may produce fewer artifacts than others, but are generally not very stable. An example of an advanced object-based MCFRC algorithm is generally disclosed in U.S. Pat. No. 6,625,333. The pixel-based methods are generally computationally expensive.
Frame rate conversion methods are also used for creating special visual effects (VFX) such as to create slow motion, fast motion or variable-motion sequences, which are frequently practiced in the production of a motion picture, commercials and video. Examples of commercial software tools available for such VFX applications include ReTimer software by Realviz and TimeWarp software by Algolith. Retimer provides the ability to create digital “slow motion” or variable motion and allows users to edit rate curves and motion vector fields to achieve desirable results. Some such commercial software tools deploy some forms of MC methods. For instance, the MC algorithms behind ReTimer are based on block image elements, while the algorithms behind TimeWarp are based on object image elements, such as those developed by Communications Research Centre Canada (CRC) and described in U.S. Pat. No. 6,625,333. Such commercial software tools, however, are not designed for automated computation, and they reply on human users to provide user inputs interactively through a GUI. Furthermore, these software tools inevitably produce unacceptable artifacts due to problems like occlusion and motion estimation errors, and they do not provide efficient tools and methods to handle those problems. Although the resulting artifacts can potentially be fixed through manual fixes by skillful human users, the process is relatively labor-intensive, costly and time consuming.
Increasing the spatial resolution of each image frame can also improve the perceived resolution of a motion picture. A conventional motion picture shot on 35 mm negative film is limited to a spatial resolution of approximately 80 cycles per mm, or approximately 1,800 lines per picture height for 1.85 projection format. Due to the generational modulation transfer function (MTF) losses from standard film lab processes, the spatial resolution of a release film print is reduced to approximately 875 lines per picture height or lower.
The advance of digital cinema technology eliminates some major sources of MTF losses, especially those from the standard film lab process, so that it becomes feasible to present a motion picture with a higher perceived resolution than a typical release film print. The DCI Digital Cinema System Specification recommends that a digital motion picture be presented at a 2K or 4K format. A 2K digital format can theoretically support a spatial resolution up to 1,080 lines per picture height, while a 4K digital format can support up to 2,160 lines per picture height. However, the quality of a digital cinema presentation cannot be guaranteed unless the quality of motion picture image content can match the spatial resolution of a digital cinema system. Because of MTF degradations from various stages of the motion picture production and post-production processes, including capture, scanning, VFX and data compression, the resulting motion picture images may have a much lower spatial resolution than what can be supported by a digital projector.
It is a major challenge to improve the spatial resolution of motion picture images in order to produce a high quality cinematic experience, especially when a motion picture is to be presented in a large format cinema. A typical large format cinema, such as an IMAX® theatre, has a screen as large as 80 feet in height. In such a theatre, the audience seated much closer to the screen than in a conventional cinema. Delivering a satisfactory visual experience to the audience in such a theatre requires significant enhancement in image quality, such as the perceived resolution. Even when such enhancement methods are applied, it is difficult to complete all required processing within a relatively short time window in order for the enhanced motion picture to be released on schedule.